A steganalysis algorithm for RGB images based on color gradient matrix was proposed in this paper. The algorithm was constructed with color gradient matrix of the RGB image, and 16-dimensional statistical features, including gradient asymmetry, gradient energy, gradient mean, gradient variance, gradient entropy and so on. Support Vector Machine (SVM) took the 16-dimensional statistical features to detect hidden message in RGB images. The experimental results indicate that the proposed algorithm realizes the reliable steganalysis of JSteg, F5, OutGuess, Steghide and MB1, which is suitable for universal steganalysis for RGB images.